The present invention relates to a method of operating image sensors according to the xe2x80x9clocal autoadaptive methodxe2x80x9d of recording highly dynamic image scenes.
Electronic image sensors are known and are manufactured in a wide variety of technologies. Such sensors are designed so that a number of picture elements (pixels) are arranged in a suitable manner, usually as a single line or as a matrix in columns and lines. An image projected onto the image sensor is converted by the pixels into an electric signal proportional to the amount of incident light at the pixel locus. The respective proportionality constant is known as the conversion sensitivity (sensitivity) of the sensor. In a very widespread type of image sensor, the xe2x80x9cintegrating image sensor,xe2x80x9d the pixels generate the output signal by integrating photoelectric currentxe2x80x2-generated charge carriers over a certain interval of time (integration time) to a capacitor. The electric signal is read out by means of suitable control signals (clock signal or read signal) applied to the pixel and readout paths leading away from the pixel and is sent with suitable means to image analyzing units or image utilizing units, such as a recording device.
An important feature of an image sensor is its dynamics, defined by the minimum and maximum image brightness at the pixel locus leading to a signal that can be utilized appropriately at the output of the image sensor. The dynamics are limited at the lower end by the electric noise of the output signal: a pixel signal smaller than the electric noise cannot be utilized. The dynamics are limited at the upper end by the saturation of the components used for signal conversion and signal transmission.
Another known problem with image sensors is that their dynamics are not sufficient to completely image the brightness contrasts occurring in many applications to the output signal. Therefore, either dark image parts are swallowed by the noise and/or lighter image parts are in saturation, which can also lead to additional interference such as smear or blooming. Several methods are known for eliminating the problem of limited dynamics.
U.S. Pat. No. 5,168,532 is cited as representative of a large number of patents and publications wherein the effective dynamics of an image sensor system are increased by the xe2x80x9cdual readoutxe2x80x9d principle. For this purpose, the image sensor system is provided with an option of varying the known sensitivity-for example, by selecting the integration time of an integrating image sensor or, even more simply, with the help of an iris diaphragm. Then two images are output by the image sensor, one at a low sensitivity and one at a high sensitivity. These two images are then combined by a suitable method to form a complete image. The method xe2x80x9cmultiple readoutxe2x80x9d which has also been patented and published in a wide variety of variants (e.g., U.S. Pat. No. 5,638,118), supplemented by the xe2x80x9cdual readoutxe2x80x9d method only inasmuch as instead of only two images, a larger number of images with different sensitivities are recorded, stored and combined. Both methods have the serious disadvantage of a complicated sensor system, which, in addition to the image sensor, also contains means for storing and processing the image data, such as a frame grabber.
A second group of methods of increasing sensor dynamics is compression of the image signal in signal generation in the pixel. With the conventional compression method of logarithmic compression, the light-dependent photoelectric currentxe2x80x2 is converted to a logarithmically dependent signal voltage by utilizing the logarithmic voltage-current characteristic of diodes or MOS transistors in operation below the threshold voltage, as published by N. Ricquier and B. Dierickx, for example, in xe2x80x9cActive Pixel CMOS Image Sensor with On-Chip Non-Uniformity Correction,xe2x80x9d IEEE Workshop on CCDs and Advanced Image Sensors, California, Apr. 20-22, 1995. These and all other compression sensors lose image details due to dynamic compression. In addition, fixed interference (so-called fixed pattern noise or FPN) which occurs in the pixels and the signal paths due to local fluctuations in component parameters such as transistor threshold voltages or capacitances is amplified exponentially, which must in turn be corrected by expensive measures. Other methods (for example, U.S. Pat. No. 5,572,074 or xe2x80x9cAnalog VLSI Phototransduction by Continuous-Time, Adaptive, Logarithmic Photoreceptor Circuitsxe2x80x9d by T. Delbruck and C. A. Mead in computation and Neural Systems Program, Technical Report, CNS Memorandum No. 30, 1994, pages 1-23, California Institute of Technology, Pasadena, Calif. 91125) control the pixel sensitivity locally through the output signal produced by the pixel itself with the help of a complicated pixel circuit; however, this method again effectively corresponds to the compression method with the same disadvantages resulting from it.
Locally adaptive methods are better methods of operating image sensors with high dynamics. A locally adaptive method is characterized in that the sensitivity of the image sensor is not adjusted for all pixels at the same time (globally) but instead is adjusted for smaller subgroups, preferably individually for each pixel (locally). According to the nature of the image sensors, it is possible to connect at different points in the signal path.
Methods that reduce, pixel by pixel, the light striking the pixel are described in U.S. Pat. No. 5,410,705 (attenuation by polarizers) and U.S. Pat. No. 5,218,485, for example. All these methods presuppose a more complex optical structure and an expensive external system for controlling the attenuation elements.
Simpler systems with higher dynamics are achieved with such locally adaptive methods which control the integration time on a pixel by pixel basis. A conventional method with locally adaptive integration time uses individual pixel reset, IPR, for example, as reported by O. Yadid-Pecht, B. Pain, C. Staller, C. Clark and E. Fossum in xe2x80x9cCMOS Active Pixel Sensor Star Tracker with Regional Electronic Shutter,xe2x80x9d in IEEE Journal of Solid-State Circuits, vol. 32, no. 2, Feb. 1997 and in xe2x80x9cWide Dynamic Range APS Star Tracker,xe2x80x9d in Solid State Sensor Arrays and CCD Cameras, San Jose, Calif., Proceedings of the SPIE, vol. 2654 (1996) pp. 82-93, and by S. Chen, R. Ginosar in xe2x80x9cAdaptive Sensitivity CCD Image Sensor,xe2x80x9d Proceedings of the SPIE, vol. 2415 (1995) pp. 303-309. With the sensors described here, the pixel circuits have been modified so that their integration capacitor can be reset individually in each pixel at any time. Due to this fact, an individual integration time can be achieved for each pixel in certain limits, and therefore the pixel sensitivity can be adapted to the light sensitivity striking the respective pixel locus. The methods described here are characterized by a low additional expense in the pixel circuit with high dynamics at the same time.
With the locally adaptive image sensor (LAS) developed at the Institute for Semiconductor Electronics at the University of Siegen, the integration time of each pixel can be programmed individually into the pixel in the form of an analog voltage before the actual integration phase (see T. Lulxc3xa9, H. Fischer, S. Benthien, H. Keller, M. Sommer, J. Schulte, P. Rieve, M. Bxc3x6ohm, xe2x80x9cImage Sensor with Per-Pixel Programmable Sensitivity in TFA Technologyxe2x80x9d; H. Reichl, A. Heuberger, Micro System Technologies ""96, VDE Verlag, Berlin, Offenbach, pages 675 ff., 1996).
A disadvantage of the IPR and LAS methods, however, is the considerable expense in terms of supplementary circuits which are needed to generate, pixel by pixel, the exposure times required for the next integration cycle and the resulting driving clock pulses from the image read out last. The additional disadvantage of the IPR methods is the problem that all pixels must be reset in the desired sequence during the integration phase, which leads to collisions and non-deterministic clock pulse overcoupling on the sensitive pixel electronics. In addition, all the adaptive methods described so far have the disadvantage of time delay; the sensitivity to be set for an integration phase is obtained from the pixel signals of the preceding integration phase, so that proper setting of pixel sensitivity according to the situation cannot be reliably guaranteed.
The invention is based on the problem of adapting the sensitivities of the pixels of an image sensor with minimal circuitry and without time delay to the brightness prevailing at the pixel locus, thereby increasing the effective dynamics of the image sensor.
This object is achieved by the adaptation to the brightness prevailing at the pixel locus by the pixel circuit itself, where the pixels are locally autoadaptive.
The principle of local autoadaptivity was made known by O. Yadid-Pecht in xe2x80x9cThe Automatic Wide-Dynamic-Range Sensor,xe2x80x9d SID 93, Intl. Symp., Digest (1993) pp. 495-498, where local autoadaptivity is achieved by the ability of the pixel to reset itself when the integrated signal exceeds a threshold. In principle, this prevents saturation of pixels, but integration time is nevertheless not adapted reliably to the light intensity striking the pixel, because it may occur, for example, that a weakly illuminated pixel is reset again just before the end of the integration phase and no mentionable signal voltage can be integrated in the very short integration time remaining.
Furthermore, the integration time selected by the pixel must be reported to subsequent image analyzing stages by a suitable method so that the brightness at the pixel locus can be reconstructed from the two parameters, the integration time and the integrated pixel signal. O. Yadid-Pecht; leaves this problem largely unanswered, and mentions only that the integration time should be stored locally in four flip-flops, which leads to a considerable expense in terms of pixel circuit and additional control lines (xe2x80x9cWidening the Dynamic Range of Pictures,xe2x80x9d High Resolution Sensors and Hybrid Systems, San Jose, Calif., Proceedings of the SPIE, volume 1656 (1992) pp. 374-382). So as not to make the, effective size of the pixel area completely uneconomical, the author also proposes that the locally autoadaptive control be implemented only once per pixel block of 8xc3x978 pixels, for example, which immediately destroys the advantage of local autoadaptivity, because a strong light-dark transition within this pixel block leads to the same restrictions on dynamics as those which occur with the globally adaptive sensors.
The solution to the problem according to this invention consists of an integrating, locally autoadaptive pixel which automatically terminates its integration time autarchically even before so many photo-generated charges have been integrated that the output signal is saturated. This method thus differs from that described above inasmuch as the pixel itself determines the end time of the integration period and not the starting time.
An advantageous method of ending the integration time is an electronic switch which is looped into the path of the photoelectric currentxe2x80x2 from the detector to the integration capacitor and which interrupts the current path from the detector to the capacitor to terminate the integration time. A method equivalent to that described above is an electronic switch which establishes a conducting connection between a first integration capacitor which the light-proportional photoelectric currentxe2x80x2 flows during the entire integration phase and a second holding capacitor at the time when the integration time of the respective pixel is to be ended so that the voltage on the holding capacitor after the end of the integration phase is proportional to the voltage on the integration capacitor at the above-mentioned time. Pixel circuits with such electric switches are adequately well known and are used in almost all integrating image sensors for global control of the integration time.
The special advantage of the solution to the problem described above is that a sensor operated in this way has much higher dynamics in comparison with a traditional sensor at a lower cost, so that this sensor supplies output signals that can be evaluated in all image segments at much higher contrasts. In addition, the output signal is always in the upper control range for a greatly expanded image brightness range, which automatically leads to an improved signal-to-noise ratio and thus an improved image quality.
An advantageous refinement of the method consists of storing information (hereinafter referred to as the integration signal) in the pixel which provides information about the length of the integration time of the pixel and is read out of the image sensor together with the integrated pixel signal so that the brightness prevailing at the respective pixel locus can be determined unambiguously from the integration signal and the pixel signal.
It is advantageous to store the integration signal in an analog form in the pixel, i.e., in the form of a voltage, a current or a charge, for example. Such analog information storage devices as a capacitor or a current level storage device are adequately well known and can be implemented in a smaller area in conventional image sensor technologies as CMOS, BiCMOS or TFT than a digital device with the same information content, as proposed by O. Yadid-Pecht, for example. However, storage of the integration signal as a digital signal can also be an advantageous option because it offers a much greater interference immunity.
In addition, there are the options of generating this integration signal autarchically within the pixel, inputting it into the pixel from the outside or generating it with the help of signals input into the pixel from the outside. The signal is preferably input into the pixel completely from the outside because in this way additional components for signal generation in the pixel are eliminated and the pixel need only store the externally applied signal. The signal to be input from the outside is suitably applied to all the pixels or at least to large groups of pixels such as entire columns or lines of pixels at the same time, so that the number of generators that must be supplied for this signal can be lower than the number of pixels which the sensor has. These generators then supply a signal such as a voltage or a current or a digital word which is preferably monotonic with the integration time already elapsed, such as a voltage ramp. As soon as a pixel has ended its integration time, it stores not only the pixel signal integrated by that time in the exposure capacitor but also the integration signal applied externally by its respective generator in its internal integration signal memory.
The special advantage of the solution to the problem described above is that the brightness information is divided into two signal parametersxe2x80x94first, the pixel signal, and second, the integration signal which provides information about the integration time. When divided into these two signals, much higher brightness dynamics can be reproduced than would be possible with a single signal, because the integration signal provides a coarse scaling, so to speak, while the pixel signal supplies detailed information. In the commercial field in particular, where numerous electric interference sources are limited to the transmission paths, this division of the dynamics represents the only possibility for processing images with very high signal dynamics ( greater than 100 dB).
An advantageous further improvement on the method consists of the fact that the integration can be ended only at a few times outside the times indicated by the pixels, which are defined by a clock signal applied to the pixel from the outside. The advantage of this discretization of time is more precise control of the integration periods that can be selected by the pixels. In particular the very short integration times established at high illuminance levels must be set and detected by the integration signal with such a high precision that the inaccuracies correspond to much shorter periods of time than the integration time per se, which is achieved in an especially advantageous manner by locked discretization of the integration times.
An advantageous further embodiment of the method, especially in conjunction with the time discretization of the integration periods and the use of an analog integration signal, is to use an amplitude-discretized analog integration signal such that the integration signal can assume only discrete amplitudes. Amplitude discretization of the integration signal has the advantage that the following image analyzing stages can reconstruct the selected integration period precisely by forming a threshold, and the interference such as noise and FPN which is unavoidable in analog circuits has no effect on the formation of a threshold if there is a sufficient distance between discretization stages and thus has no effect on the reconstructed integration time.
The case of digital storage of the integration signal naturally involves a discretization of values which corresponds to the aforementioned amplitude discretization of the analog signal. In addition, digital storage can be used most advantageously together with said time discretization, because a digital value can be assigned unambiguously to each integration time in this way. When considered from the standpoint of how it is justifiable in the most modern memory technologies for a grossly amplitude-discretized analog signal to be a multi-level digital signal, the transition from a pure binary digital signal to the amplitude-discretized analog signal is fluid and rich in variants.
An advantageous refinement of the method is the use of an exponential distribution of the discretized integration periods. An exponential distribution refers to a selection of predetermined integration periods that can be selected by the pixel such that an integration period is longer by a fixed factor than the next shorter period that can be selected. This is a distribution that is known from other applications, such as the shutter speed of a camera, for example, which ensures a uniform coverage of several powers of ten of possible light intensities.
An advantageous embodiment of the method is the technological implementation of the locally autoadaptive image sensor in the TFA technology developed by Professor Bxc3x6hm and his colleagues at the Institute for Semiconductor Electronics of the University of Siegen, as described, for example, by Professor Bxc3x6hm in xe2x80x9cIntelligent Image Sensor Systems in TFA (Thin Film on ASIC) Technology,xe2x80x9d a lecture delivered at the Special Course on Intelligent Sensors, University of Roorkee, Roorkee U. P., India, 1995. The advantage of vertical integration of the electro-optical detector on the signal processing ASIC presented there is manifested especially with locally autoadaptive sensors, because pixel electronics and detector do not mutually take each other""s places as is the case in pure CMOS technology, for example, but instead the chip area taken up by the somewhat more extensive pixel electronics is available at the same time to the detector above it, which is thus more sensitive.
Other advantageous embodiments of method may be derived from the following.
One embodiment of the invention, the preferred device for carrying out the methods described above, is shown in the following figures and described in detail below. This embodiment is an image sensor implemented in the TFA technology according to this invention with 90 dB dynamics.